Security vulnerabilities plague modern software. In computer security, vulnerability is a weakness which allows an attacker to gain an unauthorized access to the system or cause a denial of service situation for the legit users. Businesses lose money and reputation when an attack is performed. The prevent this, vulnerabilities should be fixed or mitigated as early as possible.
Vulnerabilities in software are usually caused by programming errors. Also a security risk, however, may be classified as vulnerability and there are vulnerabilities that are not related to software. Hardware, site, and personnel vulnerabilities are examples of vulnerabilities that are not software security bugs.
In security testing, a test case consists of input to a system to be tested, but in contrast to traditional testing, the expected outcome is usually not specified in detail. This is because the purpose of these tests is to locate a security flaw in the system rather than to probe some specific functionality. When a test case is executed, the tested system is subjected to specified inputs and its behavior is then monitored. A test case is considered to be a collection of machine readable and human readable data, which is machine-executable to produce a series of events.
A verdict is given for the test case on the basis of the monitored behavior. The test verdict tells whether a test has succeeded or failed. An identified security problem, such as a denial-of-service, results in a fail verdict. The absence of any unwanted behavior results in a pass verdict. When the number of test cases is large, verdicts must be automatically determined for the test cases. However, once an initial verdict of a test case is known, a problem can be further investigated by looking at the data used to determine the verdict. The test data used can be displayed on a screen. Each data set used for an individual test can consist of the input value of the individual test (a single value or a set of values), the output from the system under test and any additional information which may be collected, e.g. logs produced by the system under test.
Applications can also be tested. Black-box testing is a method of software testing that tests the functionality of an application. Specific knowledge of the application's code/internal structure and programming knowledge in general is not required. Test cases are built around specifications and requirements, i.e., what the application is supposed to do. Any system, which accepts some input can be tested by black-box testing. Black-box testing can be performed by sending messages over a network, but also by injecting files, such as images, XML files, audio files, or any other kind of files. Injection can be done also locally using any methods of input available for an application.
In software testing, fault injection is a technique for improving the coverage of a test by introducing faults to test code paths. Robustness testing (also known as Fuzzing or Fuzz testing) is a type of fault injection commonly used to test for vulnerabilities in communication interfaces such as protocols, command line parameters, or Application Programming Interfaces (APIs).
White-box testing is a method of testing software that tests internal structures or workings of an application. In white-box testing an internal perspective of the system, as well as programming skills, are required and used to design test cases.
Grey-box testing is a combination of these and the tester knows something of the internal structure in order to perform a better test.
A test suite is a collection of related test cases. An executable test suite is a test suite that can be executed by a program. The term System Under test (SUT) refers to a system that is being tested for correct operation. The term is used mostly in software testing.
Through the years, a number of different methods have been proposed for generating test cases. A test case is a description of a test and is traditionally mapped directly to and derived from use cases or be derived from system requirements.
Fuzz testing, or fuzzing, is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. In fuzzing, unexpected and/or erroneous input data is fed to the tested system. The input data can be generated in random or systematically based on definitions of the allowed input. Typically fuzz testing uses thousands or even millions of different test cases, as it is relatively cheap to produce and run a test case, as expected outcome for a test case is not defined. The difference between an expected input and a fuzz input can be called anomaly, the anomalous input being fed to the tested system. The program is then monitored for exceptions such as crashes or failing built-in code assertions. Fuzzing is commonly used to test for security problems in software or computer systems.
Modern automated security testing technique, like fuzzing, is used to find the vulnerabilities early so that they can be fixed or mitigated before they are used to attack systems. In fuzzing, test cases for security testing are created by understanding which kind of inputs are most likely to reveal vulnerabilities. Test cases are also automatically generated so that the number of test cases is high and the test cases systematically cover the input space. A tester using such a system might find critical vulnerabilities and get them fixed or mitigated.
However, often the tester neither can fix nor mitigate the problem by herself. In such cases, the tester should be able to package and deliver the information about the identified problem and how to replicate the problematic behavior to someone who can fix the problem or mitigate it by other means. There are plenty of stakeholders who would have use of getting such information, such as decision makers, quality and assurance people, network administrators, authorities, etc.
A security testing tool is a program (used by security professionals and professional hackers) that have functionality to test an application and find vulnerabilities and errors. A fuzzer test tool is an example of a security testing tool.
Currently, security testing tools provide human-readable reports which describe the found vulnerabilities. Some systems also allow creating scripts or executables, which should reproduce the vulnerability when they are executed. The reports may be located in the World Wide Web for access without the need to send e-mails etc. with the vulnerability attached.
Vulnerability management programs of identifying, classifying, remediating, and mitigating vulnerabilities exist. Setting up such programs include e.g. determining the desired security state of an environment, defining the current security state, prioritizing possible vulnerabilities, addressing causes of vulnerabilities, and monitoring and maintaining the program as an ongoing process.
Vulnerability information is collected, analyzed and shared with respect to what components are vulnerable to what kind of attacks and exploitations. For this purpose, someone first needs to find and report the problem, before it can be fixed or mitigated. Mitigation may e.g. consist of disabling or reconfiguring the vulnerable service or using a firewall configuration for blocking the attack.
Attempts have been made in categorizing vulnerabilities and ranking the severity of them in information systems but there are no common platforms for reporting and sharing vulnerability information in an efficient way. Current methods of vulnerability reporting are not standardized or suitable for co-operation. Because each producer of vulnerability reports employs a unique document structure that does not facilitate automated processing, users must manually parse individual vulnerability reports to find information that is germane to their environments.
Another problem with vulnerability reports is that it is often difficult or impossible to reproduce the sequence of events leading to the vulnerability announced in the report. Reproduction is usually required to make and test any fix or mitigation. A report may consist of a written description or a script using some scripting language.
A related problem is that a testing tool can generate millions and millions of different test cases. For regression testing of new versions of software, it is often recommended to re-run all test cases to make sure that the new version has no regression, that it has not introduced problems which were not present in the earlier version. Re-running all the test cases that originally were run against earlier versions is, however, often not feasible.